{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Object Instance Segmentation using TensorFlow Framework and Cloud GPU Technology.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s7QFZ6ztHsSC",
        "colab_type": "text"
      },
      "source": [
        "# Object Instance Segmentation using TensorFlow Framework and Cloud GPU Technology\n",
        "\n",
        "![](https://i.ibb.co/SPJWCgc/zzed.jpg)\n",
        "---\n",
        "In this guide, we will discuss a Computer Vision task: Instance Segmentation. Then, we will present the purpose of this  task  in TensorFlow Framework. Next, we will provide a brief overview of Mask R-CNN network (state-of-the-art model for Instance Segmentation). We also offer a demonstration on Mask R-CNN  model using a jupyter notebook environment: Google Colab \n",
        "\n",
        "## What is Instance Segmentation?\n",
        "\n",
        "On the one hand, the Semantic Segmentation (SS) task is one of the Computer Vision task which consists in assigning to each pixel a label among a set of semantic categories. \n",
        "\n",
        "\n",
        "\n",
        "![](https://i.ibb.co/L6TLXFQ/22E2.jpg)\n",
        "\n",
        "Ultimately, it is intended to predict a segmentation mask that indicates the category of each pixel. These pixels are classified starting from high-quality feature representations. On the other hand, Instance Segmentation (IS) is based on Semantic Segmentation techniques. It permits to recognize each object instance per pixel for each detected object. These labels are maintained by instance.\n",
        "\n",
        "The common applications and use cases that take place using the Semantic / Instance Segmentation task are the following: \n",
        "- Autonomous navigation;\n",
        "- Facial Segmentation;\n",
        "- Categorizing clothing items;\n",
        "- Precision Agriculture.\n",
        "- Etc\n",
        " \n",
        "For more details, you can look at two use cases related to Semantic Segmentation challenge:\n",
        "\n",
        "\n",
        "**Use case 1: **[Semantic Segmentation for Autonomous vehicles](https://blog.playment.io/semantic-segmentation-for-autonomous-vehicles/)\n",
        "\n",
        "**Use case 1: **[Semantic Segmentation for Facial recognition](https://blog.playment.io/improve-facial-recognition-using-semantic-segmentation-landmark-annotation/)\n",
        "\n",
        "Examples of Instance Segmentation projects and tutorials:\n",
        "\n",
        "- [Instance segmentation with OpenCV](https://www.pyimagesearch.com/2018/11/26/instance-segmentation-with-opencv/)\n",
        "\n",
        "- [ Instance Segmentation by Deep Coloring](https://github.com/kulikovv/DeepColoring)\n",
        "\n",
        "- [How to do Semantic Segmentation using Deep learning](https://medium.com/nanonets/how-to-do-image-segmentation-using-deep-learning-c673cc5862ef)\n",
        "\n",
        "\n",
        "Useful books for learning various aspects of Instance Segmentation: \n",
        "1. Practical Convolutional Neural Networks: Implement advanced deep learning models using Python: [Here](https://www.amazon.com/Practical-Convolutional-Neural-Networks-Implement/dp/1788392302)\n",
        "2. Deep Learning for Computer Vision: [Here](https://)\n",
        "\n",
        "## TensorFlow Framework for Deep Learning\n",
        "\n",
        "[TensorFlow](https://www.tensorflow.org/) is an integral open source platform for Machine Learning. It has a scalable and exhaustive environment consisting of tools, libraries and community resources that provide researchers and developers the ability to easily develop and deploy applications based on ML technology. The main features of TensorFlow are illustrated in the Figure below:\n",
        "\n",
        "![](https://d2h0cx97tjks2p.cloudfront.net/blogs/wp-content/uploads/sites/2/2018/07/Tensorflow-Features.jpg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3JYSxvKQ5H_",
        "colab_type": "text"
      },
      "source": [
        "## Prerequisites\n",
        "Before starting this guide, it is essential to be familiar with the basics of Python programming, Computer Vision concepts, Deep Learning Libraries (TensorFlow + Keras Framework), and OpenCV library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ExHZadHFRc1A",
        "colab_type": "text"
      },
      "source": [
        "## Guide map\n",
        "The content of this guide will be organized according to the following map:\n",
        "\n",
        "* What is Instance Segmentation?;\n",
        "* TensorFlow Framework for Deep Learning\n",
        "* An overview of Mask R-CNN model for Instance Segmentation;\n",
        "* Using Google Colab with GPU (enabled);\n",
        "* Mask R-CNN : Demonstration.\n",
        "* References.\n",
        " "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SMyycVuaRh6_",
        "colab_type": "text"
      },
      "source": [
        "## An overview of Mask R-CNN model for Instance Segmentation\n",
        "\n",
        "Thanks to Mask R-CNN, we can automatically segment and construct pixel masks for each object in input image. We will apply Mask R-CNN  to  visual data such as images and videos.\n",
        "Mask R-CNN algorithm was presented by He et al[1]. In fact, It builds on previous object detection works, by R-CNN (2013)[2], Fast R-CNN (2015)[3] and Faster R-CNN (2015)[4] respectively. Mask R-CNN not only generates the bounding box for a detected object, but also generates a predictive mask.\n",
        "\n",
        "Mask R-CNN model is based on Faster R-CNN architecture with 2 major contributions:\n",
        "\n",
        "1. Replacement of the ROI Pooling module by a more precise module named *ROI Align*;\n",
        "2. Inserting an additional branch from the ROI Align module.\n",
        "\n",
        "This additional branch takes the output of the ROI Align and then sends it into two  convolution layers (CONV). The output of the convolution layers (CONV) is the predicted mask itself. \n",
        "In the following figure, we can see the block diagram  of Mask R-CNN:\n",
        "\n",
        "![Mask R-CNN](https://www.pyimagesearch.com/wp-content/uploads/2018/11/mask_rcnn_mask_resizing.jpg)\n",
        "\n",
        "## Using Google Colab with GPU (enabled)\n",
        "\n",
        "Google Colab has been developped to facilitate collaboration between Machine Learning professionals in a more transparent way.\n",
        "\n",
        "Sign in to your Google Gmail account in the top right corner, if you haven't already done so. It\n",
        "will ask you to open it with Colab at the top of the screen. Then you will make a copy so that you can edit it.\n",
        "\n",
        "![](https://i.ibb.co/pzH18dw/1.png)\n",
        "\n",
        "\n",
        "It is now possible to click on \"*Runtime*\" menu button to select the Python version and use GPU/CPU device to speed up the calculation.\n",
        "\n",
        "![](https://i.ibb.co/T1JFqCf/2.png)\n",
        "\n",
        "Now, everything is ready for the environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRJJXhQmXFNY",
        "colab_type": "text"
      },
      "source": [
        "### Verification  that TensorFlow is able to detect the GPU device:\n",
        "\n",
        "Just select \"GPU\" from the Notebook Settings Accelerator drop-down menu (via Edit menu or  cmd/ctrl-shift-P command).\n",
        "\n",
        "Execute this psedo-code to confirm that TensorFlow can detect the GPU:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oh2w3SQBX4dA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "device_name = tf.test.gpu_device_name() \n",
        "if device_name != '/device:GPU:0':\n",
        "raise SystemError('GPU device is not detected') \n",
        "print('Detected GPU at: {}'.format(device_name))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qtNjNnbTYSWC",
        "colab_type": "text"
      },
      "source": [
        "It's coming out:\n",
        "\n",
        "`Found GPU at: /device:GPU:0`\n",
        "\n",
        "If you are interested in the type of GPU being used. It's a Nvidia Tesla K80 with 24G of memory. Quite powerful.\n",
        "\n",
        "Run this code to find out for yourself."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b-_yVJ8HY9Ry",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.python.client import device_lib \n",
        "device_lib.list_local_devices()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KPRodnl0ZFUP",
        "colab_type": "text"
      },
      "source": [
        "It's coming out:\n",
        "\n",
        "`physical_device_desc: \"device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7\"]`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oFfV3KU1ZKqH",
        "colab_type": "text"
      },
      "source": [
        "## Mask R-CNN : Demonstration\n",
        "\n",
        "This section provides an implementation of Mask R-CNN on Keras+TensorFlow Framework. \n",
        "\n",
        "###1. Installing dependencies and running the demo \n",
        "\n",
        "Mask R-CNN has some dependencies to install before you can run the demo. Colab allows you to install Python packages via the `pip` command, and general Linux packaging/libraries via the ` apt-get` command.\n",
        "\n",
        "In case you haven't heard yet. Your current instance of Google Colab runs on an Ubuntu virtual machine. You can execute almost all the Linux commands that you usually do on a Linux machine.\n",
        "Mask R-CNN depends on [`pycocotools`](https://pypi.org/project/pycocotools/) package, you can install it with the following commands:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6i5_N3Yjb2gI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install Cython\n",
        "!git clone https://github.com/waleedka/coco\n",
        "!pip install -U setuptools\n",
        "!pip install -U wheel\n",
        "!make install -C coco/PythonAPI"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Stzjs90b3zr",
        "colab_type": "text"
      },
      "source": [
        "It clones GitHub's repository. Install the compilation dependencies. Finally, compile and install the coco API library. All this happens in the cloud virtual machine quite quickly.\n",
        "\n",
        "You are now ready to clone the Mask R-CNN directory of GitHub and access into this directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_zLtdwWcp9X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/matterport/Mask_RCNNN\n",
        "# cd to the reference directory and possibility to download the pre-trained weight. \n",
        "import os\n",
        "os.chdir('./Mask_RCNN')\n",
        "!wget https://github.com/matterport/Mask_RCNNN/releases/download/v2.0/mask_rcnn_coco.h5"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nI1lGdSxdC9f",
        "colab_type": "text"
      },
      "source": [
        "Note that you change directories with the Python script instead of executing a `cd` shell command since you execute Python in the current notebook.\n",
        "\n",
        "Now you can run the demo of Mask R-CNN on Colab, as you would on a local machine.\n",
        "\n",
        "\n",
        "Follow the below Python codes in order to familiarize yourself with the use of a pre-trained model for detecting and segmenting objects. All psedo-codes will be commented on.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hkyHh46gekfx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#import of the necessary packages\n",
        "import os \n",
        "import sys \n",
        "import random \n",
        "import math\n",
        "import numpy as np \n",
        "import skimage.io \n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt \n",
        "import coco\n",
        "import utils\n",
        "import visualize\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1WofmwQ2etJ3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Root directory of the project \n",
        "ROOT = os.getcwd()\n",
        "# Directory to save the trained model and logs files\n",
        "MODEL= os.path.join(ROOT, \"logs\")\n",
        "# Local path to trained weights file\n",
        "COCO_MODEL = os.path.join(ROOT, \"mask_rcnn_coco.h5\") \n",
        "# Download COCO trained weights\n",
        "if not os.path.exists(COCO_MODEL): utils.download_trained_weights(COCO_MODEL)\n",
        "# Image directory to be detected\n",
        "IMAGE = os.path.join(ROOT, \"images\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pErwlnMNf5Wa",
        "colab_type": "text"
      },
      "source": [
        "###2. Model configurations\n",
        "\n",
        "We will use a model trained on the [MS-COCO dataset](http://cocodataset.org/) (It is a large-scale object detection, segmentation, and captioning dataset). The model configurations are in ` CocoConfig class` of coco.py file.\n",
        "\n",
        "Make slight changes to the configurations depending on the task. To do this, subclassify the CocoConfig class and replace its attributes that you need to modify.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0iJ4dYkSg5cI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class InferenceConfig(coco.CocoConfig):\n",
        "# Set the batch size to 1 as we will perform the inference on 1 image at a time. Batch size = GPU_NB * IMAGES_PER_GPU \n",
        "GPU_NB = 1\n",
        "IMAGES_PER_GPU = 1\n",
        "config = InferenceConfig() \n",
        "config.display()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_S0AXek0hqmK",
        "colab_type": "text"
      },
      "source": [
        "###3. Building models and importing trained weights\n",
        "\n",
        "In order to create models and load trained weights , please type the following psedo-codes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "82VyMN90iH0J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create model \n",
        "model = modellib.MaskRCNN(mode=\"inference\", model_dir=MODEL, config=config)\n",
        "\n",
        "# Load COCO trained weights\n",
        "model.load_weights(COCO_MODEL, by_name=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZcjpyPT-iYlT",
        "colab_type": "text"
      },
      "source": [
        "###4. Data preparation: MS-COCO dataset\n",
        "\n",
        "The model classifies objects and returns class IDs, which are integer values that identify each class. Some datasets assign integer values to their classes and others do not. For example, in the MS-COCO dataset, the \"person\" class is 1. IDs are often sequential, but not always. The COCO dataset, for example, has classes associated with class IDs of classes 70 and 72, but not 71.\n",
        "\n",
        "To get the list of class names, you can load the dataset and then use the `class_names` property like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kGP3iqPhjBsY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Loading MS-COCO dataset\n",
        "dataset = coco.CocoDataset() \n",
        "dataset.load_coco(COCO_DIR, \"train\")\n",
        "dataset.prepare()\n",
        "# Print class names\n",
        "print(dataset.class_names)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGHQAwAKjRir",
        "colab_type": "text"
      },
      "source": [
        "You have included the list of class names below. The name index of the class in the list represents its ID (first class is 0, second is 1, etc.)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2KqxRF7rjlxE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# COCO Class names by indexes\n",
        "class_names = ['BG','person','bicycle','car','motorcycle','airplane','bus','train','truck','boat','traffic light']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TWRH2AiBj-TA",
        "colab_type": "text"
      },
      "source": [
        "###5. Starting object detection process\n",
        "\n",
        "To perform object detection, just type the following psedo-codes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dryw6gJrkPkB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Loading a random image from the dataset\n",
        "\n",
        "file_names = next(os.walk(IMAGE))[2]\n",
        "image = skimage.io.imread(os.path.join(IMAGE, random.choice(file_names)))\n",
        "# Running object detection\n",
        "results = model.detect([image], verbose=1)\n",
        "# Evaluating results \n",
        "r = results[0]\n",
        "visualize.display_instances(image, r['kings'], r['masks'], r['class_ids'],\n",
        "class_names, r['scores'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ubLoZzuvlIKL",
        "colab_type": "text"
      },
      "source": [
        "###6. Customization of images to be segmented\n",
        "\n",
        "You can download an image from a third party website such as:\n",
        "\n",
        "- [Imgbbbb](https://imgbbb.com/)\n",
        "- [GitHub](https://github.com)\n",
        "\n",
        "You can download your image using `wget` command.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "isUQbRBblt8K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Loading a random image from the dataset\n",
        "file_names = next(os.walk(IMAGE_DIR))[2]\n",
        "image = skimage.io.imread(os.path.join(IMAGE,'my_image.jpg'))\n",
        "# Running object detection\n",
        "results = model.detect([image], verbose=1)\n",
        "# Evaluating results \n",
        "r = results[0]\n",
        "visualize.display_instances(image, r['kings'], r['masks'], r['class_ids'],\n",
        "class_names, r['scores'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTrSeup9mN8K",
        "colab_type": "text"
      },
      "source": [
        "For example, the result of object detection and segmentation is shown below:\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/1200/1*8eIcVM-M506P4bA0Y7pbag.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TvmswGRNmwIz",
        "colab_type": "text"
      },
      "source": [
        "### 7. Video object segmentation\n",
        "\n",
        "There are 3  steps to processing a video file.\n",
        "\n",
        "\n",
        "1. Transforming video frames into static images;\n",
        "2. Image processing;\n",
        "3. Converting processed images into output videos.\n",
        "\n",
        "In our previous demo, we asked the model to process only 1 image at a time, as configured in `IMAGES_PER_GPU` option."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LvhNTSXWoMSZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class InferenceConfig(coco.CocoConfig):\n",
        "#  Set the batch size to 1 as we will perform the inference on 1 image at a time. Batch size = GPU_NB * IMAGES_PER_GPU \n",
        "IMAGES_PER_GPU = 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nV7e6XeqoU9e",
        "colab_type": "text"
      },
      "source": [
        "If we are going to process all the video at once, it will take a long time. We will therefore use the GPU to operate several frames simultaneously.\n",
        "The Mask R-CNN pipeline is quite computationally intensive and requires a lot of GPU memory. In Colab, The Tesla K80 GPU with 24G of memory can safely process 3 images at a time. If you go any further, the notebook may crash in the middle of video processing.\n",
        "Thus, in the psedo-code below, we set the ` batch_size` to 3 and use the `cv2 library` to take 3 images at a time before processing them with the model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CxeiWbrVpIp6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "capture = cv2.VideoCapture(os.path.join(VIDEO, 'demo.mp4')) \n",
        "while True:\n",
        "ret, frame = capture.read()\n",
        "# Save each frame of the video to a list\n",
        "frame_count += 1\n",
        "frames.append(frame)\n",
        "if len(frames) == batch_size:\n",
        "results = model.detect(frames, verbose=0)\n",
        "for i, item in enumerate(zip(frames, results)): frame = item[0]\n",
        "r = item[1]\n",
        "frame = display_instances(\n",
        "frame, r['kings'], r['masks'], r['class_ids'], class_names, r['scores']\n",
        ")\n",
        "name = '{0}.jpg'.format(frame_count + i - batch_size) \n",
        "name = os.path.join(VIDEO_SAVE_DIR, name)\n",
        "cv2.imwrite(name, frame)\n",
        "# For starting the next batch \n",
        "frames = []"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kLGk-3yMqFeb",
        "colab_type": "text"
      },
      "source": [
        "After running this psedo-code, you should now have all the processed image files in `./videos/save folder.`\n",
        "The next step is easy, you have to generate the new video from these images. We will use `VideoWriter ()` function from OpenCV (cv2) to do this.\n",
        "\n",
        "But there are two things you want to be sure of:\n",
        "\n",
        "**1. Images must be indexed in the same way**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tZ9NiNaoqq9D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get all image file paths.\n",
        "images = list(glob.iglob(os.path.join(VIDEO_SAVE,' *.*'))\n",
        "# Sort the images by index.\n",
        "images = sorted(images, key=lambda x: float(os.path.split(x)[1][:-3]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zveb1kbWrAwK",
        "colab_type": "text"
      },
      "source": [
        "**2. The frame rate corresponds to the original video. You can use the following psedo-code to check it or simply open the file property.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mTz7DryBrQgz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "video = cv2.VideoCapture(os.path.join(VIDEO_DIR, trailer1.mp4'));\n",
        "# Get OpenCV version\n",
        "(major_ver, minor_ver, subminor_ver) = (cv2. version).split('.'')\n",
        "if int(major_ver) < 3 :\n",
        "fps = video.get(cv2.cv.CV_CAP_PROP_FPS)\n",
        "print(\"Frames per second: {0}\".format(fps)) else :\n",
        "fps = video.get(cv2.CAP_PROP_FPS)\n",
        "print(\"Frames per second: {0}\".format(fps))\n",
        "video.release();"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N4cFHtElrroi",
        "colab_type": "text"
      },
      "source": [
        "Finally, you can use this psedo-code to generate video from the processed images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YhHx3vlXr3PD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def generate_video(outvid, images=None, fps=30, size=None,is_color=True, format=\"FMP4\"):\n",
        "  \n",
        "  \"\"\"\n",
        "Create a video from a list of images.\n",
        "@param outvid output video\n",
        "@param images list of images to use in the video \n",
        "@param fps frame per second\n",
        "@param size size of each frame \n",
        "@param is_color color\n",
        "\n",
        "\"\"\"\n",
        "from cv2 import VideoWriter, VideoWriter_fourcc, imread, resize \n",
        "fourcc = VideoWriter_fourcc(*format)\n",
        "vid = None\n",
        "for image in images:\n",
        "if not os.path.path.exists(image):\n",
        "  raise FileNotFoundError(image)\n",
        "img = imread(image) \n",
        "if vid is None:\n",
        "  if size is None:\n",
        "size = img.shape[1], img.shape[0]\n",
        "vid = VideoWriter(outvid, fourcc, float(fps), size, is_color)\n",
        "  if size[0] != img.shape[1] and size[1] != img.shape[0]:\n",
        "img = resize(img, size) \n",
        "vid.write(img)\n",
        "vid.release() \n",
        "return vid"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8YkUTlnnsxAs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import glob import os\n",
        "# Image directory to be detected\n",
        "ROOT = os.getcwd()\n",
        "VIDEO = os.path.join(ROOT, \"videos\")\n",
        "VIDEO_SAVE = os.path.join(VIDEO, \"save\")\n",
        "images = list(glob.iglob(os.path.join(VIDEO_SAVE, '*.*'))) \n",
        "# Sort the images by index\n",
        "images = sorted(images, key=lambda x: float(os.path.split(x)[1][:-3]))\n",
        "outvid = os.path.join(VIDEO, \"out_video.mp4\") \n",
        "generate_video(outvid, images, fps=30)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bpMhDs4Etea2",
        "colab_type": "text"
      },
      "source": [
        "Once this step is completed, the segmented video should now be ready to be downloaded into your local machine."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dTSrXbg-thkT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files \n",
        "files.download('videos/out_video.mp4')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UWYja1LGvxt1",
        "colab_type": "text"
      },
      "source": [
        "##List of References:\n",
        "[1] K. He, G. Gkioxari, P. Dollár, and R. Girshick, \"Mask R-CNN\", arXiv:1703.06870[cs], March 2017.\n",
        "\n",
        "[2] R. Girshick, J. Donahue, T. Darrell, et J. Malik, « Rich feature hierarchies for accurate object detection and semantic segmentation », arXiv:1311.2524 [cs], nov. 2013.\n",
        "\n",
        "[3] R. Girshick, \"Fast R-CNN\", arXiv:1504.08083[cs], Apr. 2015.\n",
        "\n",
        "[4] S. Ren, K. He, R. Girshick, et J. Sun, « Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks », arXiv:1506.01497 [cs], juin 2015.\n",
        "\n",
        "####Other sources :\n",
        "TensorFlow, https://www.tensorflow.org/\n",
        "\n",
        "Keras, https://keras.io/"
      ]
    }
  ]
}
